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Introduction 

The first decade of the 21st century is swiftly experiencing an increased need for learning 
that has greatly exceeded present delivery capacity of traditional educational settings. To 
help meet this need, many educational institutions are now offering courses as well as 
entire degree programs through distance education. Thus, institutions are thrust into being 
confronted with a rapidly changing social as well as economic scenario; and if they are to 
survive, they must evolve. Crucial to this successful evolution is the formulation of new 
assessment strategies for distance education (Stallings, 2002). Because of the process and 
practice of assessment which increasingly has become the focus of the public, politicians, 
and legislators, future historians may view the 1980s as the decade in which problems in 
education achieved national significance, and the 1990s as the decade of accountability 
(Seward, in Stallings, 2002). 

The federal government, accrediting agencies, new technologies, as well as complaints 
from colleges about the accreditation process are all forces driving the push to reevaluate 
how we assess what students learn. The problem is how do we meet such broad mandates? 
Because distance education is relatively new, critics hold it to a higher standard as opposed 
to traditional education. Thus, since distance education has more to prove, educators are 
trying to establish a variety of assessments that prove its effectiveness (Ewell, in 
Carnevale, 2001). Accrediting agencies are encouraging colleges to move beyond the 
traditional measures of success such as satisfaction surveys from students, employers, and 
alumni. They stress the use of a variety of measurement tools to audit students' work such 
as examining their writing and critical-thinking skills. However, many faculty dislike the 
notion of measuring what students have learned and some find the idea that surveys are 
more meaningful than grades insulting. It has been compared with the idea of faculty 
being or not being able to receive tenure based on the number of publications they have. 
Other faculty believes measurement is useful as long as it is not at the expense of more 
important issues (McMurtrie, 2000). 

Stallings (2002) asserts that published statements by accreditation commissions and other 
influencers indicate that there is an increased emphasis being placed on accountability in 
learning as well as the outcomes of the educational process. One such agency is the 
Commission on Colleges of the Southern Association of Colleges and Schools (SACS). SACS 
is tasked with accrediting institutions of higher education. They ensure that institutions 
meet established regional standards. At the heart of the Commission's philosophy of 
accreditation is the concept of institutional effectiveness. Institutions must demonstrate 
how well they fulfill their purpose and must engage in an ongoing quest for quality. They 
must provide documentation of the quality and effectiveness of their programs. Conducting 
self-studies to evaluate established standards of quality and the strength of the institution 
and to determine that educational goals are being met is a way of addressing the issue of 
effectiveness. Institutions are expected to use their purpose statement as the basis of 
planning and evaluation, to use a variety of assessment methods, and to demonstrate use 
of the results to improve educational programs. Additionally, both quantitative and 
qualitative data that demonstrate student achievement should be used when evaluating an 
academic program ("Criteria for Accreditation," 1998). 

Purpose 
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Through external accreditation review processes, university self-study assessments, and 
student reports of self-learning, it is evident that student learning in the traditional face- 
to-face setting is demonstrated and validated through research. However, as more and 
more institutions of higher education are transcending the traditional boundaries to offer 
courses as well as entire degree programs on line, the issue of distance education 
effectiveness has spawned vigorous debates. Thus, this study will seek to identify multiple 
assessment strategies that can be used to evaluate the effectiveness of distance education 
courses within a degree program. 



Review of Literature 

McMurtrie (2000) identifies one common thread among accreditors— they are all revising or 
developing better ways for colleges to focus on and measure student learning. These 
accrediting agencies included the Middle States Association of Colleges and Schools, New 
England Association of Schools and Colleges, North Central Association of Colleges and 
Schools, Northwest Association of Schools and Colleges, Southern Association of Colleges 
and Schools, and the Western Association of Schools and Colleges, Commission for 
Community and Junior Colleges. Traditionally, within the classroom, a variety of means are 
used by effective teachers to determine how well students are learning. They use tests, 
quizzes, exams, research papers and homework to formally evaluate student achievement 
and to assign grades. To informally evaluate learning, teachers ask questions, listen to 
comments and questions from students, and monitor facial and body expressions. Such 
evaluations allow teachers to adjust their teaching by slowing down when necessary to 
review material or to clarify questions, confusion, and misunderstandings. They also allow 
teachers to move forward when students exceed expectations. However, different 
challenges face teachers who must teach at a distance. Teachers are no longer surrounded 
by a familiar, traditional classroom. They lack contact with a homogeneous group of 
students as well as face-to-face feedback such as student questions, body language, facial 
expressions, and comments. Thus, distance educators must not only formally evaluate 
students, but must also use an informal approach to collect data that determines teaching 
effectiveness, clarity of course content, appropriateness of assignments, and student 
comfort with the methods of delivery of instruction ("Guide #4," n.d.). 



Carnevale (2001) notes that the difference between distance assessment and traditional 
classroom assessment is that distance education programs for the most part are geared 
toward those already in the workforce. Since this involves learning by doing, students 
complete projects to show that they can apply what they have learned in addition to 
understanding it. Being able to apply what has been learned is a focus of many assessment 
policies. He also notes that some universities have created their own assessment exams, 
and some buy them from organizations such as the Educational Testing Service and ACT. 
These assessment exams may take the form of essays, multiple-choice questions, or 
projects whereby students must demonstrate their knowledge. A project might include 
simulations of a real world job. For example, students enrolled in information technology or 
network administration have to plan computer networks for small companies. The plan is 
presented in a five-page essay detailing financial and technical requirements for 
implementing the plan, similar to what an employee would present to the manager of a 
company. Students enrolled in a business science program have to create a marketing plan 
for a small company. In a five-page essay, students specify how a company can improve its 
contact with customers. Such projects are a part of the students' portfolio requirement. 
The portfolios are reviewed to assess whether students have demonstrated their 
knowledge and skills to earn a degree. To ensure that students are able to work 
collaboratively once they enter the workforce, some assessment strategies include having 
students work in a collaborative fashion to create a product instead of just learning by 
lecture. In addition, the University of Phoenix expanded the assessment practices of its 
face-to-face program to its fast-growing distance education program. Both traditional and 
on-line students take the same 100 multiple-choice question tests at the beginning of their 
academic program and at the end. The data is then analyzed to see how much students 
have learned from the program. Further, projects are given that assess students' critical 
thinking skills, communication skills, as well as cognitive abilities. 
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THE STUDY 



Population 

The study focuses on a Business, Career, and Technical Education Department within a 
College of Education at a large doctoral university in eastern North Carolina. The 
department offers 1) undergraduate teacher education programs: BSBE in Business 
Education, BSBE in Marketing Education, and BSBE in Business and Marketing Education; 2) 
undergraduate non-teaching programs: BSBE in Information Technologies and Information 
Technologies (Community College 2+2 Community Partnership); and 3) graduate 
programs: MAEd in Business Education and Marketing Education, and an MS in Information 
Technologies. At the time this study was conducted, the department also offered a degree 
in family and consumer sciences education. This program has since relocated to the Human 
and Environmental Sciences Department. 



Methodology 

To evaluate the effectiveness of the distance education program in information 
technologies, comparisons of three courses being taught simultaneously on line and in the 
classroom were made to see if significant differences existed in student outcomes 
assessments. Students within these courses were assigned specific tasks to be evaluated. 
The criterion established for these courses was reviewed by two or more independent 
raters. The classes included financial information systems II, business communications, 
and programming with Microsoft visual basic. In addition, most of the programs within the 
department are offered in the classroom as well as on line, many of them simultaneously. 
Thus, the same assessment measures used for SACS for the traditional classroom were 
equated with the distance education classes for those in the field of Family and Consumer 
Sciences, Business Education, the BE/ME program, the MAEd program in Technical 
teaching, and Information Technologies. 

ASSESSMENT OF CLASSES 
Financial Information Systems II Class 

Traditional and distance education accounting students were required to produce an 
accounting simulation using Peachtree Accounting software. Ninety percent of the students 
were required to attain a mean rating of 80% or higher on a 100-point rating scale on an 
audit exam of the simulation. Two independent raters evaluated the 1) Accuracy of 
Transaction Entries, 2) Accuracy of Printed Formal Reports, 3) Completion of all Reports, 
and 4) Accuracy of the Audit Exam. Below are the results for the traditional campus section 
and for the distance section. The scale used for this assessment was: Scale: 1 = poor 2= 
fair 3 = good 4 = very good 5 = excellent. 
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Assessment Report for 
Financial Information Systems II 


CAMPUS SECTION 




Evaluator #1 


Evaluator #2 


Average of 
Evaluator 1 & 2 


1 


46 


(92) 


5 


(100) 


4.8 


(96) 


2 


46 


(92) 


44 


(88) 


4.5 


(90) 


3 


5 


(100) 


5 


(100) 


5 


(100) 


4 


3.8 


(76) 


3.6 


(72) 


3.7 


(74) 


5 


4.8 


(96) 


4.6 


(92) 


4.7 


(94) 


80% of the students sampled attained a mean 


rating of 4 (80%) or higher 




D/STANCE SECTION 




Evaluator #1 


Evaluator #2 


Average of 
Evaluator 1 & 2 


1 


4.2 


(84) 


4.8 


(96) 


4.4 


(88) 


2 


5 


(100) 


5 


(100) 


5 


(100) 


3 


46 


(92) 


44 


CO 

CO 


4.5 


(90) 


4 


4 8 


(96) 


4.6 


(92) 


4.7 


(94) 


5 


4.8 


(96) 


4.8 


(96) 


4.8 


(96) 


100% of students sampled attained a mean rating of 4 (80%) or higher 



Business Communication Class 

Business communication students were required to work collaboratively in groups of three 
to produce a research paper on international business communication that portrayed their 
expertise in the researching of international business communications as a final project. 
Criteria for success was that 90% of the students would attain a mean rating of 4.0 or 
higher on a 5 point rating scale when the papers were assessed by two independent raters. 



Research papers were evaluated for content, use of APA writing style, mechanics, 
grammar, and punctuation. The results revealed the following: 



SCALE: 1 = poor 2= fair 




3 = good 


4 = good 


5 = excellent 


Items evaluated: 

Content 
APA Style 

Mechanics, grammar, and punctuation 


Student Reports 


Evaluator #1 


Evaluator #2 


Average of 
Evaluator 1 & 2 


Traditional Vs. 
Campus 


1 


4.9 (98) 


4.9 (98) 


4.9 (98) 


Traditional 


2 


4.9 (98) 


5 


(100) 


4.9 (99) 


Traditional 


3 


5 (100) 


5 


(100) 


5 (100) 


Traditional 


1 


4.5 (91) 


5 


(100) 


4.8 (96) 


On-line 


2 


4.7 (94) 


5 


(100) 


4.8 (97) 


On-line 


3 


4.6 (93) 


5 


(100) 


4.8 (97) 


On-line 


100% of students sampled attained a mean rating of 4 (80%) or higher. 



Programming with Microsoft Visual Basic Class 
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Students in a basic programming for business applications course had to produce a 
computer software program that portrayed their expertise in the design and programming 
of a visual basic computer program. Criteria for success were that 90% of the students 
would attain a mean rating of 4.0 or higher on a 5 point rating scale. 



SCALE: 1 = poor 2= fair 3 = good 4 = good 5 = excellent 



Items Evaluated: 

The program runs and all forms are named correctly. 

Buttons and menu items exist and work properly. 

The graphical appearance /layout is pleasing and follows "Windows standards. 

The calculation form functions correctly. 

The album form displays a picture and label when user makes a selection from the 
specified list. 





Distance Class 


Traditional Class 


Evaluator #1 


91.2% 


92.8% 


Evaluator #2 


92.8% 


96.8% 


Average Score 


92.0% 


94.8% 



ASSESSMENT OF PROGRAMS 

There are certain requirements that all students who graduate from the program must 
meet, whether taking courses by distance or traditionally. The same standard formal 
assessment measures used in traditional classes to satisfy SACS requirements were also 
used in distance education classes. Examples include items such as: 



Business Education BSBE 


1) Intended Student Outcomes 


Candidates completing the program will be able to 
demonstrate knowledge of their specialty areas in the 
field of Business Education at a level sufficient to 
obtain licensure in North Carolina 


Means of Assessment 


Candidates will take the PRAXIS Exam #10100 in their 
Senior I or Senior II semester. 100% of the candidates 
will attain a passing score of 580 (established by the 
North Carolina Department of Public Instruction) or 
higher on their first attempt and no category score will 
be below 60%. 


2) Intended Student Outcomes 


Candidates completing the BE/ME program will be able 
to develop a portfolio that documents their teaching 
skills. (Note: The Council for Teacher Education (CTE) 
at ECU instituted a portfolio requirement for graduates 
which was effective in Spring 1999.) 


Means of Assessment 


Candidates will submit a teaching portfolio at the end 
of their Senior II semester. 95% of the candidates will 
submit portfolios and 95% of the portfolios will contain 
an instructional unit, behavior management plan, and 
evidence of technology skills as verified by the 
university supervisor and Office of Clinical Services 
staff. 
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Marketing Education BSBE 



Intended Student Outcomes 


Candidates completing the program will be able to 
demonstrate knowledge of their specialty areas in the 
field of Marketing Education at a level sufficient to 
obtain licensure in North Carolina 


Means of Assessment 


Candidates will take the PRAXIS Exam #10560 in their 
Senior I or Senior II semester. 100% of the candidates 
will attain a passing score of 690 (established by the 
North Carolina Department of Public Instruction) or 
higher on their first attempt and no category score will 
be below 60%. 



Information Technologies BSBE 


Intended Student Outcomes 


Candidates completing the Information Technologies 
program will obtain employment positions in business 
and industry. 


Means of Assessment 


From an annual survey of graduates, graduates will 
report employment data to the Department. 100% of 
the candidates seeking employment will report that 
they are employed when the annual survey is made of 
BVTE IT graduates. 



All students, traditional and distance were required to meet the same set of criteria as that 
listed above. Samples of this information are posted and can be found at: 

http:/ 7www.ecu.edu/ pir/ARBS/AcademicUnits/Business%20Education,%20B.S.B.E.htm. 

Finally, the Family and Consumer Sciences program also had specific criteria identified for 
both traditional and distance education students. First, students in this program were 
required to produce a traditional paper 1040 and a computer generated 1040 income tax 
return. A criterion for success was that 90% of the students would successfully pass the 
audit component on their first attempt. In each of the cases above, both traditional and 
distance students met the criteria for success. There was no significant difference between 
the two groups. Second, the students had to develop an early experience manual that 
documented 16 hours of clinical experience in the FACs program area. The criterion for 
success was that 95% of the manuals would contain observation logs, reflections, and 
classroom analysis when evaluated by a team consisting of the university professor, the 
clinical teacher, and a teacher educator from the BVTE Department. Both groups met the 
criteria for success. Third, students were required to participate in a minimum of one one- 
day professional development activity. Criteria for success was that 95% of the students 
would attend and document the professional activities participated in through a reflection 
when evaluated by a team consisting of three faculty members in the BCTE department. 
Both groups met the criteria for success. 

CONCLUSION 

The assessment measures provided identified multiple assessment strategies that can be 
used to evaluate the effectiveness of distance education courses within a degree program. 
These strategies included projects, research papers, portfolios, qualitative, and 
quantitative data. Results of these measures revealed no significant difference in 
achievement between traditional campus and distance education students. 
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